Less Is More: Rejecting Unreliable Reviews for Product Question Answering

Promptly and accurately answering questions on products is important for e-commerce applications. Manually product (e.g. community question platforms) results in slow response does not scale. Recent studies show that reviews are a good source real-time, automatic (PQA). In the literature, PQA formulated as retrieval problem with goal to search most relevant answer given question. this paper, we focus issue of answerability reliability using reviews. Our investigation based intuition many may be answerable finite set When answerable, system should return nil answers rather than providing list irrelevant reviews, which can have significant negative impact user experience. Moreover, questions, only included result. We propose conformal prediction framework improve systems, where reject unreliable so returned more concise accurate at question, including returning unanswerable questions. Experiments widely used Amazon dataset encouraging our proposed framework. More broadly, demonstrate novel effective application methods task.